2-class Internal Cross-validation Pruned Eigen Transformation Classification Trees

In [3] it has been demonstrated that decision trees built in a feature space yielded by some eigen transformation can be competitive with industry standards. Unfortunately, the selection of such a transformation and the dimension of the feature space that should be retained is not self-evident. These trees however have interesting properties that can be exploited. Since the order of the splits is fixed due to the known importance of each feature given by their corresponding eigenvalues, all trees are pruned versions of the largest tree. This property makes it possible to prune such a tree based on an internal cross-validation using the training data. This allows us to use a technique that should overfit less than for example the estimated error rates used in C4.5 classification trees for pruning [4], while still using the entire training data to build the tree. We therefore present an algorithm that divides the training data into folds similar to a cross-validation. The split values are calculated for each of the obtained internal training folds. The nodes of the tree are then evaluated with the corresponding internal test folds and nodes that overfit are pruned. This is done using each of the eigen transformations. The best tree is selected and the final split values are then calculated for the selected pruned tree based on the entire training data. Results show that we can expect trees that are optimal or near optimal if there is enough training data relative to the size of the tree.